What you see is what you (don't) get: a comment on Funke's (2014) opinion paper

In an opinion paper published in this journal, Funke (2014) argues that two different types of assessment instruments for complex problem solving (CPS), computer-simulated microworlds (CSMs), and minimal complex systems (MCSs), might require different types of causal cognition. CPS denotes the ability to successfully deal with new, intransparent, and dynamically changing problem situations (Funke, 2001) and is considered one of the most important skills of the 21st century. Given the recent attention CPS has received from both academic and educational stances, for instance, through the Programme for International Student Assessment, which tested CPS in 15-yearold students across more than 40 countries (OECD, 2014), the topic addressed by Funke (2014) is both timely and of high relevance. In this commentary, we will elaborate on the difference between specific CPS assessment instruments used in a variety of research fields (e.g., experimental psychology, educational assessment) and CPS as the underlying attribute, and we will offer a view that diverges from Funke’s (2014). We will express our hope that different CPS assessment traditions that are reflected in CSMs and MCSs will converge toward a generalizable understanding of CPS as an unobserved latent attribute (i.e., a psychological concept) that is of relevance to researchers from a number of fields including experimental psychology and individual differences research. Before alluding specifically to Funke’s opinion paper, we would like to specify our terminology. Funke uses the terms CPS and MCS for the two types of assessment. However, CPS denotes a psychological attribute and not a specific set of assessment instruments. Thus, we will use the established terms computersimulated microworlds (CSMs) and minimal complex systems (MCSs) for the assessment instruments, and we will reserve the term complex problem solving (CPS) for the latent attribute that both types of assessments claim to tap into. Funke (2014) argues that the usefulness of CSMs and MCSs for measuring CPS depends on the field of study. CSMs, with their realistic and highly complex setup, their many interrelated variables, and their knowledge-rich semantic embedding, are best applied in experimental settings, whereas MCSs, with an analytical approach that is geared toward reducing complexity and a high level of standardization, are best suited for (educational) assessment purposes. This position is reflected in the predominant use of CSMs in experimental and cognitive research (e.g., Dörner, 1980) and of MCSs in (educational) assessment (e.g., Wüstenberg et al., 2012). On the basis of task analyses, Funke claims that the type of causal cognition, the heuristics, and the strategies required in CSMs and MCSs differ substantially and, thus, do not allow for direct comparisons between the two. For any study, it is of crucial importance that the employed measures—be they CSMs or MCSs—tap into the construct they claim to capture even though the measures may differ with regard to their difficulty or their surface features. Admittedly, CSMs and MCSs look quite different at first sight. CSMs are knowledge-rich and simulate complex real-world scenarios such as business companies (e.g., the Tailorshop) or entire cities (e.g., Lohhausen), whereas MCSs are knowledge-lean and less complex. But can we really conclude, on the basis of purely conceptual task analyses, that they tap into different types of causal cognition? And if so, what is it that they tap into? 15 years ago, Süß (1999) had already provided important empirical guidance on this question with regard to the construct validity of CSMs. He showed that the performances in three different CSMs were moderately correlated—which was to be expected given that they supposedly all measured CPS. More importantly, when controlling for fluid intelligence and specific prior knowledge, the correlation dropped to non-significance. In the terms of contemporary theories on the human intellect (McGrew, 2009), the shared variance of three CSMs originated from fluid intelligence and (specific) prior knowledge. That is, the type of causal cognition required to successfully master CSMs is empirically identical to the causal cognition required for standard tests of fluid intelligence. Provocatively, one could state that the Tailorshop, probably the best-known microworld included in Süß’s study, requires nothing but fluid intelligence and inductive reasoning as well as specialized knowledge about how companies work. Following this, the strong impact of context in CSMs and the substantial advantages for participants who possess this specific knowledge are not only assets of CSMs that make them more realistic but also disadvantages that distort the measurement of the underlying attribute and the cognitive processes